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Abstract 

Space  and  Naval  Warfare  Systems  Center,  San  Diego’s  (SSC-SD)  Command  21  project, 
sponsored  by  ONR,  is  addressing  how  information  technology  can  be  designed  to  best  facilitate 
information  production,  consumption,  and  management.  For  the  past  several  years,  the  focus  of 
the  Command  21  effort  has  been  the  development  of  Knowledge  Web  (K-Web),  which  utilizes 
Web  technologies  to  share  operationally  relevant  information.  In  K-Web,  data  is  processed  and 
stored  by  ‘producers’  in  a  way  that  represents  meaningful  knowledge  to  ‘consumers’.  Use  and 
utility  of  K-Web  at  the  Global  2000  war  game,  were  reported  at  last  year’s  CCRTS.  K-Web  was 
implemented  on  USS  Carl  Vinson  in  May  2001,  for  use  during  deployment.  Upon  the  ship’s 
return,  interviews  of  users  were  conducted,  focusing  on  use  and  utility  of  K-Web  tools,  products, 
business  rules  and  training  materials.  The  interview  data  indicate  K-Web  as  invaluable  for 
asynchronous,  distributed  dissemination  of  operational  information.  Additionally,  automatically- 
collected  data  were  analyzed  for  patterns  of  use.  Quantitative  and  qualitative  data  enabled 
assessment  of  how  the  current  K-Web  concept  and  technologies  support  users’  requirements 
within  operational  environments.  These  analyses  were  also  compared  to  analyses  from  the  war 
game  environment. 

Introduction 

As  technology  increases  the  speed  and  ease  of  information  exchange,  the  problems  associated 
with  information  production,  consumption,  and  management  also  increase.  Effective  command- 
level  decision-making  requires  a  high  level  of  situation  awareness  (SA)  (Endsley,  1995).  To 
support  improved  SA,  information  must  be  made  available  in  a  format  that  is  meaningful  and 
useful  for  application  to  real-world  problems;  in  other  words,  information  must  be  transformed 
into  knowledge.  Space  and  Naval  Warfare  Systems  Center,  San  Diego’s  (SSC-SD)  Command  21 
project,  sponsored  by  ONR,  is  addressing  how  information  technology  can  be  designed  to  best 
facilitate  information  production,  consumption,  and  management  in  support  of  the  needs  of 
senior  decision  makers  and  their  staff  in  military  command  centers. 


For  the  past  several  years,  the  focus  of  the  Command  21  effort  has  been  the  development  of  a 
concept  referred  to  as  Knowledge  Web  (K-Web).  The  K-Web  concept  utilizes  Web-based 
technologies  to  share  operationally  relevant  information.  In  K-Web,  available  data  is  processed, 
formatted,  and  stored  by  ‘information  producers’  in  a  way  that  represents  meaningful  knowledge 
to  the  ‘information  consumers’.  A  number  of  tools  have  been  developed  to  facilitate  the 
formatting,  dissemination  and  Web-based  presentation  of  information  that  is  easily  shared  and 
understood.  The  tools  and  products  were  designed  to  meet  14  user  requirements  determined 
through  a  cognitive  task  analysis  (Smallman,  Oonk  &  Moore,  2001).  The  initial  design  and 
implementation  of  these  tools  was  in  an  operational  exercise  environment  at  the  Global  2000  war 
game.  Analyses  of  the  usage  and  utility  of  the  K-Web  in  this  environment  were  reported  at  last 
year’s  CCRTS  (Moore  &  Averett,  2001;  Oonk,  Smallman  and  Moore,  2001).  Several  changes 
were  made  to  the  K-Web  tools  and  products  based  on  those  analyses  and  follow-up  laboratory 
experiments  (Oonk,  Smallman,  Obermayer,  Pester-Dewan  &  Rogers,  2001) 

The  first  operational  implementation  of  the  K-Web  occurred  following  the  Global  2000  war 
game  onboard  the  US  S  Carl  Vinson  in  May  2001,  for  use  during  their  WESTPAC  deployment 
(see  Table  1).  COMCARGRU  THREE  (CCG3)  and  DESRON9  were  embarked  aboard  the  USS 
Carl  Vinson  and  used  K-Web  during  the  deployment.  The  USS  Carl  Vinson  Battle  Group  arrived 
in  the  North  Arabian  Gulf  on  September  11,  2001.  Subsequently,  CCG3  became  the  Composite 
Warfare  Commander  (CWC)  for  military  activities  supporting  Operation  Enduring  Freedom. 
CCG3  and  his  staff  used  a  functional  area  based  concept  of  operations  for  Command  and 
Control,  which  was  supported  by  K-Web.  This  combination  of  organizational  structure  and 
information  technology  made  CCG3  uniquely  qualified  to  take  on  the  CWC  role.  Having  K-Web 
aboard  allowed  the  Battle  Group  to  innovate  and  explore  the  implications  of  Web-enabled  Speed 
of  Command. 


May  2001 

K-Web  /  K-Desk  installation  onboard  USS  Carl  Vinson 

Jul 2001 

USS  Carl  Vinson  /  CCG3  deploys 

Sep  2001 

Terrorist  attack  on  World  Trade  Center  /  Operation  Enduring  Freedom  begins 

Jan  2002 

USS  Carl  Vinson  /  CCG3  returns  from  deployment 

March  2002 

Automated  Data  Retrieved  /  Observational  Data  Collected 

Table  1.  Timeline  of  K-Web  on  USS  Carl  Vinson  WESTPAC  deployment. 

Following  the  ship’s  deployment,  interviews  of  K-Web  users  were  conducted  (March,  2002). 

The  focus  of  these  interviews  was  the  use  and  utility  of  the  K-Web  tools,  products,  business  rules 
and  training  materials.  In  addition,  data  collected  automatically  by  the  K-Web  software  were 
analyzed  in  order  to  determine  patterns  of  use  over  the  course  of  the  deployment.  Quantitative 
and  qualitative  data  analyses  enabled  the  assessment  of  how  current  K-Web  concept  and 
technologies  support  users’  requirements  within  critical  operational  environments. 


Implementation  of  K-Web  on  the  USS  Carl  Vinson 

The  following  information  is  intended  to  provide  a  brief  overview  of  the  K-Web  concept  and 
tools,  as  they  were  implemented  aboard  the  USS  Carl  Vinson,  as  well  as  at  the  Global  2001  war 
game.1 

In  K-Web,  multiple,  distributed  “information  producers”  create  Web  pages,  by  inputting 
available  information,  tactical  pictures  and  other  resources  into  a  standardized  Web  page 
template  called  SumMaker  (Figure  1).  SumMaker  products  are  referred  to  as  “Summary  Pages” 
(Figure  1).  Each  functional  area  within  a  command  produces  a  Summary  Page  that  contains 
operationally  relevant  information  pertaining  to  their  area.  These  Summary  Pages  are  all  linked 
through  a  common  “Overview  Page”  (Figure  1),  which  serves  as  both  an  integrated  status  tool 
for  quick-look  SA  and  as  a  navigational  tool.  The  “information  consumer”  is  then  able  to  view 
the  Overview  Page  and  linked  Summary  Pages  from  either  a  single-monitor  PC,  a  multiple- 
monitor  PC  (Knowledge  Desk)  or  a  large  three-screen  display  (Knowledge  Wall).  Figure  2 
represents  the  information  flow  within  a  K-Web.  Table  2  shows  which  user  requirement(s)  each 
feature/tool  was  designed  to  meet. 


For  more  detailed  information  on  this  topic  see  Moore  &  Averett  (2001). 


1 


SumMaker  Summary  Page  Overview  Page 


Figure  1.  SumMaker  (left)  has  a  “drag-and-drop”  and  pull-down  menu  interface.  “Producers”  fill  in 
fields  with  headlines  then  link  to  qualifying  information.  Summary  Pages  (center)  are  SumMaker 
products.  Users  drill  down  for  information  via  links.  Overview  Page  (right)  gives  integrated  status 
information  and  serves  as  a  navigation  tool  for  accessing  Summary  Pages.  See  Table  2  re:  numbers. 


Information  /  Knowledge  Producers 


Information  /  Knowledge  Consumers 


Other  Info 


Knowledge  Wall  Display  Users 


Figure  2.  K-Web  information  flow.  Information  producers  (left)  create  Web  pages  that  are 
accessible  to  information  consumers  (right)  from  multiple  display  types.  See  Table  2  re:  numbers 


# 

Feature 

User  Requirements 

# 

Feature 

User  Requirements 

1 

Functional  Area  Title 

Consistency 
*Situation  awareness 

7 

Alerts  and  Impacts 

Consistency 

*Supplemental  information 
Cognitive  support 
*Drill  down 

2 

Date  /  Time  Stamp 

information  age  /  reliability 

8 

Related  Links 

Consistency 

*Supplemental  information 
*Drill  down 

3 

Status  Titles 

Consistency 

9 

Status  Bar 

integrated  Information 
*Quick-look  SA 

4 

Diamonds  /  Colors 

integrated  information 
Cognitive  support 

10 

Summary  Page  Links 

integrated  Information 

5 

Graphic  Area 

intuitive  graphical  interface 

11 

K-Desk  / 

K-Wall 

*Shared  SA 
integrated  Information 
Connectivity/Collaboration 
*Flexible  Configuration 

6 

Producer  Email 

information  age  /  reliability 

Table  2.  K-Web  features  and  tools  were  designed  to  meet  specific  user  requirements. 


Method 

Assessment  of  how  the  K-Web  concept  and  technologies  support  users’  requirements  within 
critical  operational  environments  is  possible  using  both  quantitative  and  qualitative  data 
produced  by  the  use  of  K-Web  during  Operation  Enduring  Freedom.  These  data  were  collected 
using  multiple  methods:  automated,  interview  and  survey.  Two  groups  of  users  aboard  the  USS 
Carl  Vinson  were  targeted,  CCG3  and  DESRON9.  These  two  groups  had  both  common  and 
unique  user  requirements.  For  example,  although  each  group  used  K-Web  for  information 
dissemination,  the  purpose  of  information  sharing  and  the  target  audiences  varied  significantly 
across  the  two  groups2.  This  fact  facilitated  the  capture  of  different  perspectives  and  use  of  K- 
Web. 

K-Web  software  applications  automatically  captured  and  stored  several  types  of  quantitative 
data,  resulting  in  a  historical  record  of  1)  publish  rates;  2)  use  of  specific  software  features;  and 
3)  structure  and  content  of  top-layer  Web  pages.  These  data  were  collected  from  each  of  eight 
functional  areas  (FAs):  Air  Defense,  C3,  CWC,  Force  Protect,  Maritime  Ops,  MetOc,  ROE-JAG 
and  Schedules3  (henceforth  referred  to  as  FA1-FA8,  though  not  respectively). 

Upon  returning  from  WESTPAC  deployment,  qualitative  data  were  collected  by  interviewing  19 
users  on  an  individual  basis  (four  users  were  interviewed  in  pairs).  All  users  were  both  producers 
and  consumers  of  K-Web  content,  representing  seven  functional  areas:  CWC,  Intel,  JAG, 
Logistics,  Maritime  Ops,  MetOc  and  TLAM.4  Interview  questions  focused  on  the  use  and  utility 
of  the  K-Web  tools,  products,  business  rules  and  training  materials.  Additional  qualitative  data 
were  collected  via  an  anonymous  on-line  survey  addressing  these  same  topics,  using  both 
multiple  choice  and  open-ended  questions.  The  seven  respondents  represent  two  functional  areas: 
Maritimes  Ops  and  Air  Defense  (some  survey  respondents  were  also  interview  respondents). 

The  same  versions  of  the  K-Web  products  and  tools  had  previously  been  evaluated  with  regard 
to  use  and  utility  during  the  Global  2001  war  game  (July,  2001).  Results  of  automated  and 
observational  data  collected  from  the  Global  2001  war  game  provide  additional  perspectives  and 
suggestions  for  the  status  and  development  of  the  K-Web  concept  and  tools. 

Results  and  Discussion 

“Having  multiple  people  who  are  not  on  watch  and  not  at  the  same  place,  all  having  access  to 
the  same  information  is  invaluable.  It  [K-Web]  is  a  fantastic  tool  that  didn’t  become  clear  until 
Operation  Enduring  Freedom  broke  out....  This  was  an  operator’s  dream,  ” 
(Commodore,  Maritime  Operations,  DESRON9). 

1.  K-Web  use  and  access 


2  CCG3  staff  created  information  primarily  for  own  use  and  for  use  by  other  large  ships  and  commands 
with  high  bandwidth.  DESRON9  staff  typically  created  information  products  for  use  by  smaller  ships 
with  lower  bandwidth. 

3  Command,  Control  and  Communications  (C3);  Composite  Warfare  Commander  (CWC);  Maritime 
Operations  (Maritime  Ops);  Meteorology  and  Oceanography  (MetOc);  and,  Rules  of  Engagement  -Judge 
Advocate  General  (ROE-JAG). 

4  Intelligence  (Intel)  and  Tomahawk  Land  Attack  M  issile(TLAM). 


Although  the  intended  audience  of  K- Web’s  content  resided  aboard  USS  Carl  Vinson,  others 
throughout  the  Battle  Group,  other  commands  and  entities,  such  as  the  Pentagon,  were  accessing 
the  USS  Carl  Vinson  K-Web.  The  automated  data  show  that  during  the  time  period  of  September 
29,  2001,  to  December  29,  2001,  K-Web  was  accessed  by  an  average  of  482  visitors  per  day 
(range  =  340  to  67 6). 5  Output  from  automated  data  capture  shows  a  clear  pattern  of  high  and  low 
usage  point  throughout  the  day,  with  5  a.m.  -  8  a.m.  and  3  p.m.  -  6  p.m.  being  the  peak  hours  of 
use.  All  survey  respondents  indicated  that  they  used  K-Web  seven  days  a  week  during 
deployment,  with  use  ranging  from  1-14  hours  per  day.  The  CCG3  group  never  used  PowerPoint 
to  brief,  in  the  traditional  sense,  during  deployment.  During  interviews,  users  described  how  K- 
Web  was  used  to  conduct  briefings  to  senior  officers  instead  of  using  lengthy  PowerPoint6  slide 
presentations.  Through  K-Web,  commanders  had  access  to  current  information  at  all  times',  as  a 
result,  briefing  sessions  were  used  for  directly  discussing  the  issues  and  problem  solving,  while 
using  K-Web  as  a  backdrop.  Users  agreed  that  K-Web  was  useful  as  a  compliment  to  other 
tactical/real-time  tools. 

When  asked  whether  or  not  it  was  difficult  to  implement  K-Web  into  their  organization,  CCG3 
felt  that  it  was  an  “easy”,  “natural”  transition.  DESRON9  had  a  somewhat  difficult  transition, 
due  to  the  lack  of  training  and  technical  problems  (unrelated  to  K-Web).  Both  groups  responded 
that  direction  and  support  from  the  commanders  were  the  critical  factors  that  made  transition  to 
K-Web  possible  and  successful. 

2.  SumMaker  Software  ( information  producer  perspective) 
a.  Ease  of  use 

Information  producers  used  SumMaker  software  to  create  and  update  Summary  Pages.  All 
CCG3  users  stated  that  they  did  not  have  previous  Web  publishing  experience,  yet  found 
SumMaker  easy  to  learn  and  use.  The  features  of  the  software  met  most  of  their  needs,  however 
almost  every  functional  area  expressed  the  need  for  more  flexibility  in  the  SumMaker  template. 
For  example,  many  areas  did  not  follow  the  Today/Tomorrow/Long-Range  headline  titles7,  and 
desired  the  flexibility  to  change  the  number  of  fields  available  for  headlines.  One  functional  area 
expressed  the  need  for  several  different  SumMaker  templates,  each  with  a  different  focus: 
graphic,  statistical,  or  text.  Survey  responses  reflect  similar  information,  with  users  rating  the 
Alerts  &  Impacts  fields  as  needing  improvement. 

Due  to  limited  access  to  the  software  and  computers  systems,  most  DESRON9  users  did  not  use 
SumMaker.  One  person  who  did  use  SumMaker  indicated  that  it  was  easy  to  use. 


5  A  single  IP  address  might  represent  an  individual  person  or  it  might  represent  a  router  or  firewall 
behind  which  many  individuals  might  be  "hidden."  Therefore,  it  is  possible  that  the  actual  number  of 
persons  visiting  using  the  K-Web  is  significantly  higher.  Determining  the  number  of  actual  "visitors  per 
day"  is  extremely  difficult  and  is  based  on  an  elaborate  formula  taking  into  account  unique  I P  addresses, 
domains,  and  discernibly  separate  visits  or  sessions.  Due  to  a  number  of  technical  considerations,  these 
numbers  may  not  be  enti  rely  accurate.  They  do,  however,  give  a  general  sense  of  the  number  of  people 
who  visited  the  K-Web. 

6  PowerPoint®,  Word®,  Chat  and  Excel  ©are  registered  trademark  of  the  Microsoft  corporation. 

7  These  titles  were  included  in  thedesign  based  on  results  of  cognitivetask  analyses. 


All  survey  respondents  that  used  SumMaker  indicated  that  the  tool  “usually”  or  “always”  met 
their  needs  as  information  producers. 

b.  Color  and  Diamond  Use 

Colored  status  indicators  were  used  on  Summary  Pages  to  indicate  the  severity  of  impact 
information  had  on  the  mission  plan  (red  indicating  most  severe,  yellow  indicating  less  severe 
and  green  indicating  no  impact).  Analysis  of  the  data  collected  automatically  by  SumMaker 
shows  that  most  (82%)  published  status  indicators  were  green  (Figure  3).  The  dominant  use  of 
green  reflects  adherence  to  business  rules:  colors  were  being  used  to  indicate  the  impact  on 
mission  plan,  not  the  severity  of  the  situation  at  large.  Change  in  status  was  not  indicated  often: 
3%  of  status  indicators  reflected  degraded  status  (e.g.  green  to  yellow)  and  2%  reflected 
improved  status  (e.g.  red  to  yellow),  relative  to  previous  published  status  (Figure  4). 


Figure  3.  The  use  of  status  colors  on  Summary  Pages  over  the  fifteen-week  period  (left)  and  for 
the  eight  functional  areas  assessed. 


Change  in  Status 


Figure  4.  Changes  in  status  reflected  on  Summary  Pages. 

Diamonds  were  included  on  summary  pages  to  alert  the  consumer  of  “new”  or  “changed” 
information  (squares  indicated  that  information  had  not  changed  recently).  On  average,  a 
Summary  Page  displayed  1.47  diamonds  (Figure  5).  Only  4%  of  pages  published  reflected  that  a 
diamond  had  either  been  added  or  removed  (when  compared  to  the  previously  published  page) 
(Figure  6).  When  producing  information,  most  users  relied  on  colors  and  diamonds  to  alert  others 
of  new  or  important  information,  unless  the  information  was  particularly  urgent,  in  which  case 
they  would  use  the  colors  and  diamonds  in  addition  to  more  traditional  methods,  like  Chat.6 


Inclusion  of  Change  Diamonds  on  Summary  Pages 
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Figure  5.  The  use  of  diamonds  on  Summary  Pages. 


Figure  6.  Changes  in  shape  reflected  on  Summary  Pages. 


c.  Linking  and  Cross-linking 

Summary  pages,  on  average,  contained  13.61  links  to  other  K-Web  content.  This  number  was 
consistent  across  most  areas  (Figure  7).  Linked  information  varied  in  format  with  over  1 1 
different  formats  being  commonly  used.  The  most  commonly  used  formats  across  all  areas  being 
Web  pages,  graphics  and  PowerPoint  (Table  3). 


Number  of  Links  Per  Page 


Figure  7.  The  number  of  links  included  on  Summary  Pages. 
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Table  3.  The  format  of  links  provided  on  Summary  Pages  by  Functional  Area  (as  percentage  of 
total  links).6 

Several  users  found  cross-linking  (i.e.  providing  a  link  to  a  product  created  by  another  functional 
area)  to  other  areas  useful.  Producers  interpreted  cross-links  to  their  own  page  from  other  area 
pages  as  an  indication  that  their  content  was  useful.  One  significant  problem  that  cross-linking 
presented  was  that  there  was  no  way  for  a  producer  to  know  if  the  information  they  cross-linked 
to  had  changed,  other  than  by  continuously  monitoring  that  link’s  content,  which  was 
cumbersome.  Another  problem  with  cross-linking  occurred  when  the  author  of  the  cross-linked 
material  moved  the  material  into  a  different  folder,  which  broke  the  path  to  link.  Despite  these 
problems,  all  SumMaker  users  who  responded  to  the  survey  rated  cross-linking  as  “usually”  or 
“always”  helpful  when  creating  K-Web  content. 

d.  Update  Rates 

The  average  daily  update  rates  for  the  summary  pages  derived  from  the  automated  data  are 
shown  in  Figure  8.  On  average,  each  area  published  1.58  summary  pages  a  day;  this  rate  varied 
considerably  over  time  and  across  functional  areas.  In  the  interviews,  each  represented  area 
indicated  they  updated  at  least  once,  and  often  several  times  a  day.  Updating  was  driven  by  many 
factors,  including  1)  the  rate  of  information  coming  in,  2)  the  needs  of  consumers,  3)  mission 
requirements  and  4)  consumer  feedback.  For  example,  several  users  reported  that  off-ship 
consumers  notified  them  if  information  was  not  updated  quickly  enough,  which  helped  set  the 
pace.  Survey  respondents  and  interviewees  agreed  that  information  was  updated  in  K-Web  at  a 
rate  that  met  their  needs.  Often,  underlying  K-Web  content  was  updated  more  frequently  than 
summary  page  data. 


Figure  8.  Average  daily  Summary  Pages  update  rates. 


One  concern  expressed  by  several  users  from  one  group  was  the  time  invested  in  keeping  K-Web 
current.  Time  spent  updating  pages  ranged  from  an  hour  to  several  hours  a  day.  Although  several 
stated  that  maintaining  K-Web  was  time  consuming  from  the  production  side,  they  felt  it  was 
more  than  made  up  for  on  the  consumer  side.  Survey  respondents  and  interviewees  agreed  that 
posting  information  to  K-Web  was  less  time  consuming  than  previous  methods  used  to  update 
the  Commander  and  others.  (There  is  no  baseline  to  compare  time  taken  to  disseminate  similar 
kinds  of  data  using  traditional  methods.)  The  general  consensus  was  that  K-Web  content  was 
nearly  “real-time”  and  therefore  could  be  used  as  an  operational  tool. 

e.  Feedback  About  Content: 

When  asked  what  drove  page  content,  most  responded  that  feedback  from  consumers  was  a 
primary  consideration.  Along  with  including  content  that  fit  the  mission  requirements  based  on 
their  judgment  and  experience,  content  was  a  reflection  of  questions,  requests  and  other 
communications  from  consumers.  Survey  respondents  indicated  that  conversations,  emails  and 
Chat  with  watchstanders  and  other  functional  areas,  along  with  direction  from  senior  officers 
contributed  to  deciding  what  to  include  on  a  page.  These  interactions  also  contributed  to  an 
understanding  of  what  content  that  was  produced  was  actually  useful.  Some  users  noted  a 
significant  drop  in  the  number  of  questions  coming  in  on  a  particular  topic  once  that  topic  was 
posted  to  their  web  site.  Feedback  also  helped  information  producers  to  know  what  context  to  put 
their  information  in.  Producing  content  for  one’s  own  functional  area  use  usually  required  less 
contextual  or  background  information  than  producing  content  for  users  outside  the  group.  This 
led  to  users  outside  the  group  sometimes  asking  for  more  clarification.  Thus,  questions  about 
interpretation  and  consequences  of  the  information  presented  on  their  Summary  Page  influenced 
content  elaboration.  The  influence  of  feedback  on  page  content  resonated  throughout  all  areas 
aboard  USS  Carl  Vinson.  When  asked  if  it  was  important  to  know  who  had  seen  their  page  and 
when,  the  consistent  answer  across  survey  and  interview  respondents  was  that  it  was  not 
important  to  know  this  information;  it  was  assumed  that  the  right  people  had  seen  it.  Specifically, 
they  would  not  want  any  kind  of  report  or  automated  device  alerting  them  of  who  had  seen  their 
page  and  when. 

f.  Training  and  BRs 

Most  CCG3  users  participated  in  K-Web  training  in  one  or  both  training  sessions  conducted  by 
Command  21  developers  during  Global  2000  or  aboard  ship  during  K-Web  installation.  These 
users  found  that  both  learning  to  use  K-Web  and  training  newcomers  to  use  K-Web  was  quick 
and  easy.  Although  users  trained  during  the  initial  sessions  were  exposed  to  training  materials 
and  “quick  reference  guides”,  interview  respondents  reported  that  these  were  not  necessary  to 
use  when  training  newcomers. 

CCG3  users  explained  that  common  business  rules,  based  on  tool  functionalities  and  capabilities 
learned  in  training,  were  developed  during  initial  work-ups  of  the  deployment  and  were  adhered 
to  throughout  deployment.  These  business  rules  designated  that  a  diamond  on  a  summary  page 
indicated  new  information  and  that  status  colors  pertained  to  the  severity  of  impact  information 
had  on  the  mission  plan.  Business  rules  for  use  of  diamonds  and  colors  in  their  use  on  the  high- 
level  links  of  Summary  Pages  seemed  consistent  across  areas.  One  functional  area  used 
diamonds  differently  at  lower-level  links.  For  example,  colors  corresponded  with  preset 
thresholds  of  supply  availability.  These  indicators  were  intended  for  internal  use,  so  that  while 


red  diamond  at  a  lower-level  link  was  an  alert  within  this  area,  it  did  not  necessarily  require  the 
Admiral’s  attention.  Therefore,  red  alerts  at  a  lower-level  link  were  not  always  reflected  as  red 
alerts  in  higher-level  links.  Another  common  business  rule  was  that  K-Web  would  be  updated 
consistently  and  used  as  a  briefing  tool,  in  lieu  of  the  traditional  PowerPoint  presentation  (which 
was  the  previous  standard).  One  business  rule  changed  by  the  users  pertained  to  the  titles  used  in 
the  left  three  links  of  the  Summary  Page.  Originally  designed  to  represent  Today,  Tomorrow  and 
Long-Range  status,  functional  areas  each  adopted  their  own  status  titles  for  these  fields. 
Chronological  organization,  although  useful  to  some,  was  not  optimal  for  all  areas. 

Most  DESRON9  users  did  not  receive  any  training  from  Command  21  developers  and  were  not 
aware  of  training  materials  available  on-line.  Lack  of  training  resulted  in  many  users  not 
knowing  what  diamonds  and  colors  meant.  Because  most  users  only  produced  and  updated 
information  products  within  lower-level  links  and  did  not  work  with  SumMaker,  they  were  able 
to  ignore  the  colors  and  diamonds  if  they  did  not  understand  them.  Approximately  half-way  into 
the  deployment,  most  people  used  and  understood  these  alerts  in  the  same  way  that  CCG3  did.  In 
contrast  to  CCG3,  DESRON9  users  still  produced  PowerPoint  briefs  and  simply  attached  them  to 
their  Web  page.  One  business  rule  that  was  not  established  within  this  group  was  a  maximum  file 
size  for  pictures  and  attachments.  DESRON9  was  directing  content  to  off-ship  consumers  with 
low  bandwidth  capacity.  These  consumers  would  often  be  unable  to  access  content  they  needed 
due  to  large  file  sizes. 

One  issue  that  consistently  challenged  CCG3  or  DESRON9  staffs  was  who  to  produce  content 
for.  In  other  words,  should  the  content  be  directed  to  the  Admiral,  USS  Carl  Vinson  only,  the 
entire  Battle  Group  or  beyond?  Each  producer  seemed  to  cater  to  a  different  audience  depending 
on  feedback  they  received,  old  habits  or  personal  opinion  about  how  the  tool  should  be  used.  In 
response  to  the  survey  question  of  who  content  should  be  directed  toward,  users’  answers  varied 
from  other  battle  group  units  to  shore  sites  to  coalition  partners.  Some  survey  and  interview 
respondents  thought  content  should  be  directed  only  at  ownship.  The  combination  of  survey  and 
interview  results  reveals  an  important  issue  with  regard  to  content  production. 

3.  Summary  Pages  ( information  consumer  perspective) 
a.  Color  and  Diamond  Use 

Users  from  CCG3  found  diamonds  and  colors  helpful  for  identifying  important  or  changed 
information.  Survey  and  interview  respondents  agreed  that  keeping  track  of  new  information  in 
K-Web  was  usually  easy.  Both  groups  explained  that  although  colors  and  diamonds  would 
sometimes  draw  their  attention  and  influence  K-Web  navigation,  for  the  most  part,  users  would 
consistently  monitor  the  pages  they  were  most  interested  in,  regardless  of  colors  and  diamonds. 
Many  users  from  DESRON9  did  not  find  colors  and  diamonds  useful  during  the  first  half  of 
deployment  because  they  were  not  aware  of  how  to  use  them  or  what  they  meant.  Even  after 
establishing  a  business  rule  for  diamond  use  and  meaning,  this  group  tended  to  rely  more  on 
established  methods  of  alerting,  such  as  phone,  Chat  or  face-to-face.  Colors  and  diamonds  did 
lose  meaning  when  an  area  repeatedly  used  the  same  color  with  a  link.  Both  groups  expressed 
the  need  for  a  timestamp  linked  to  colors  and  diamonds  for  more  precise  information  age. 


b.  Preference  /  Ease  of  Use 


In  contrast  to  the  stated  need  for  more  flexibility  in  the  SumMaker  template  by  information 
producers,  several  users  indicated  that  having  a  standardized  format  was  useful  as  a  consumer  for 
navigation  purposes.  In  addition,  the  structured  and  limited  space  in  the  Summary  Pages  forced 
users  to  publish  only  relevant  and  needed  content,  which  resulted  in  survey  respondents  and 
interviewees  agreeing  that  they  were  usually  quickly  and  easily  able  to  locate  the  mission-related 
information  they  needed  within  K-Web.  Survey  respondents  also  reported  that  information  was 
usually  presented  formats  that  met  their  needs  without  further  clarification. 

4.  Knowledge  Wall/Desks 

a.  Usage  of  K-Wall -  briefing  and  SA 

Users  from  both  groups  recognized  great  added  value  gained  by  using  Knowledge  Desks  (6- 
headed  displays  provided  for  information  consumption).  The  multi-headed  display  helped  them 
monitor  K-Web  pages,  multiple  Chat  rooms  and  email,  while  also  working  on  MS  Office 
applications.  Regardless  of  which  applications  were  being  viewed,  several  interview  respondents 
determined  that  the  value  that  K-Desk  offered  was  the  ability  to  monitor  multiple  tasks  at  once. 
The  K-Wall,  known  to  CCG3  users  as  the  “Video  Wall”  in  TFCC,  and  the  SmartBoard  in  their 
briefing/war  room,  proved  useful  for  different  reasons.  Again,  regardless  of  the  content  being 
viewed,  the  K-Wall  allowed  several  people  to  simultaneously  view  and  discuss  the  same 
information.  Several  users  stated  that  the  combination  of  K-Web  and  K-Wall  facilitated 
collaboration  during  traditional  briefing  times,  as  opposed  to  one-way  presentations. 

b.  Legibility 

Of  the  survey  and  interview  respondents  that  had  access  to  K-Wall  or  a  K-Desk,  all  indicated 
having  no  problem  viewing  information  from  either  display  type. 

5.  Overview  Page 

a.  Integrated  status 

Several  users  from  both  groups  indicated  that  because  each  area  used  different  headings  for  the 
Today/Tomorrow/Long-Range  fields  that  the  status  bars  on  the  Overview  Page  did  not  have 
consistent  meaning.  Despite  this,  the  status  bar  was  still  used  for  quick-look  status  information; 
six  out  of  seven  survey  respondents  rated  the  status  bar  as  either  “acceptable”,  “very  good”  or 
“excellent”  with  respect  to  their  needs  as  information  consumers. 

b.  Navigation 

Survey  and  interview  agreed  that  locating  the  information  they  needed  was  usually  easy  and  that 
organization  by  functional  areas  worked  well.  Some  users  relied  on  the  Overview  Page  as  a  table 
of  contents  to  the  different  areas,  while  others  set  up  shortcuts  that  linked  directly  to  the  pages 
they  consistently  accessed.  Navigation  within  the  areas  was  easy  for  most  users  because  the  same 
kind  of  information  was  generally  stored  and  updated  in  the  same  place  on  the  Summary  Page. 
Some  areas  did  store  considerably  more  information  than  other  areas  under  single  headlines, 
which  made  it  more  difficult  to  navigate.  When  this  problem  arose,  users  would  simply  ask  the 
producers  of  that  page  where  to  find  needed  information. 

Users  from  DESRON9  suggested  that  a  search  or  find  function  would  be  helpful,  not  only  for 
finding  a  specific  piece  or  type  of  information,  but  also  for  recognizing  relationships  between 
pieces  of  information.  Navigation  within  their  site  seemed  more  difficult  because  they  had 


several  functional  areas  under  the  heading  Maritime  Ops,  resulting  in  a  smaller  K-Web  within  a 
larger  K-Web ’s  structure. 

6.  Training 

When  asked  what  improvements  would  a)  make  K-Web  easier  to  use,  or  b)  enhance  its 
functionality,  DESRON9  users  responded  unanimously  that  training  would  be  the  single  most 
important  improvement.  This  group  did  not  participate  in  any  training  sessions  (with  the 
exception  of  one  individual),  nor  were  they  aware  of  the  on-line  training  materials.  Without 
training,  users  relied  on  learning  from  each  other  and  through  trial  and  error.  This,  along  with 
limited  access  to  SumMaker,  a  rapidly  changing  information  environment,  and  some  technical 
problems  (unrelated  to  K-Web),  caused  frustration  among  users  during  the  first  few  months  of 
deployment.  Once  everyone  was  “up  to  speed”  on  how  to  use  K-Web,  the  value  was  recognized, 
but  training  could  have  alleviated  many  problems  up  front.  Specifically,  DESRON9  users 
recommended  help  functionality  embedded  in  the  software  tools.  CCG3’s  experience  with  K- 
Web,  however,  did  not  lead  to  the  same  suggestions.  In  general,  CCG3  users  expressed  that  the 
training  they  received  was  good  and  that  training  newcomers  took  a  matter  of  minutes  (answers 
ranged  from  5-30  minutes).  This  group,  however,  did  participate  in  training  at  the  Global  2000 
war  game  and  aboard  ship,  during  K-Web  installation.  It  may  be  that  newcomers  are  so  easily 
trained  because  they  are  starting  in  an  established  K-Web  environment  (Goldstein,  1993). 
Although  exhaustive  help  functionality  may  be  useful,  or  even  required  for  a  command  that  picks 
up  K-Web  without  training,  this  not  the  recommended  solution.  Training  the  command  prior  to 
installation  is  more  desirable,  economical  and  effective.  Once  the  majority  of  people  are 
comfortable  with  using  the  basic  K-Web  concept  and  tools,  newcomers  and  veterans  may  benefit 
from  on-line  help  assistance  (Koslowski  &  Salas,  1997).  Clearly,  releasing  the  software  products, 
without  either  training  or  on-line  support  could  be  a  costly  waste  of  time  and  operational 
capabilities. 

Conclusions  and  Recommendations 

The  concept  of  Web-based  information  production  and  dissemination  to  facilitate  speed  of 
command  was  tested  aboard  the  USS  Carl  Vinson  during  Operation  Enduring  Freedom.  Overall, 
K-Web  proved  to  be  an  extremely  valuable  tool  for  the  production  and  dissemination  of 
information  in  this  critical  operational  environment. 

The  major  lessons  learned  from  the  evaluation  of  the  usage  of  K-Web  aboard  the  USS  Carl 
Vinson  are  listed  below.  A  similar  evaluation  was  conducted  of  the  same  K-Web  tools  and 
products  when  they  were  being  used  at  the  Global  2001  war  game  (see  Oonk,  Rogers  &  Moore, 
2002  for  more  details).  Whenever  applicable,  findings  that  emerged  from  that  evaluation  that 
augment  or  contrast  with  those  from  the  “real-world”  K-Web  evaluation  are  also  included. 

Major  lessons  learned: 

•  Usage  aboard  the  USS  Carl  Vinson  suggests  that  training  needs  to  occur  prior  to 

implementation  of  the  K-Web  concept  and  tools,  and  needs  to  be  supplemented  with  help 
functionality; 

o  Similarly,  the  Global  2001  evaluation  suggested  a  need  for  integrated  training  to 
promote  a  better  understanding  of  K-Web  tool  capabilities  by  information 
producers  and  consumers. 


•  Business  rules  regarding  tool  use  and  content  audience  need  to  be  established  prior  to 
implementation; 

o  Similarly,  the  need  for  mechanisms  for  improved  dissemination  of  K-Web 

business  rules  to  support  optimal  use  of  the  K-Web  by  information  providers  and 
consumers  emerged  from  Global  2001. 

•  Accessibility  of  K-Web  content  by  external  consumers  is  impacted  by  file  sizes;  and 

•  Different  functional  areas  have  different  content  organization  needs,  which  implies  the 
need  for  more  flexibility  in  the  templates. 

o  Global  2001  also  suggested  a  similar  need  for  tools  and  content  that  are  flexible 
and  customizable  to  meet  the  needs  of  different  users. 

The  keys  to  USS  Carl  Vinson’s  success  using  K-Web  were: 

•  Easy  to  learn,  easy  to  use  tools; 

o  This  makes  K-Web  easy  to  learn  and 
o  keeps  file  sizes  small. 

•  Functionalities,  capabilities  and  designs  that  met  users’  needs; 

o  this  contrasts  with  the  findings  from  Global  2001  which  suggested  a  need  for 
better  cognitive  tools  to  support  cognitive  functionality,  such  as  attention 
management  and  change  alerting. 

•  Combination  of  consumer  and  producer  driven  content; 

o  this  contrasts  with  the  identified  need  for  improved  feedback  mechanism  to 
enhance  content  production  that  emerged  strongly  from  the  artificial  war  game 
environment. 

•  Establishment  and  adherence  to  business  rules; 

•  Consistent  formatting  across  functional  areas; 

•  Self-determined  update  rates;  and 

•  Leadership  support. 
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